<html>
<head><meta charset="utf-8"><title>ast::tokenstream::TokenStream parsing · clippy · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/index.html">clippy</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html">ast::tokenstream::TokenStream parsing</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="246842540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246842540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246842540">(Jul 22 2021 at 12:57)</a>:</h4>
<p>Btw is there a way to parse the ast::tokenstream::TokenStream to _ast_ and visit it? E.g. I want to get all identifiers or values used there, probably in some naive way.</p>



<a name="246843175"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246843175" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246843175">(Jul 22 2021 at 13:02)</a>:</h4>
<p>Could also be to HIR..</p>



<a name="246844126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246844126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> xFrednet <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246844126">(Jul 22 2021 at 13:11)</a>:</h4>
<p>Rust provides visitors for this. I've only used them in HIR but it might be possible for AST tokens as well. Here is an example visitor that collects all used values: <a href="https://github.com/rust-lang/rust-clippy/blob/610381455cb78a31695b94bc45ee580cf0cd7b38/clippy_lints/src/copies.rs#L535-L589">example</a>. I'm not sure if there is a nicer way at the moment, it also depends on your use case <span aria-label="upside down" class="emoji emoji-1f643" role="img" title="upside down">:upside_down:</span></p>



<a name="246847769"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246847769" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> flip1995 <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246847769">(Jul 22 2021 at 13:40)</a>:</h4>
<p>I'm getting the feeling that you try to implement a global analysis in Clippy. Can you please provide some context, what you're trying to do? I don't say that you aren't allowed to work on something like that, I just want to prevent, that you put much work into something that won't get accepted in the end.</p>



<a name="246873774"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246873774" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246873774">(Jul 22 2021 at 16:50)</a>:</h4>
<p>xFrednet: yeah, I've done something very similar too</p>



<a name="246873880"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246873880" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246873880">(Jul 22 2021 at 16:51)</a>:</h4>
<p><span class="user-mention" data-user-id="264664">@flip1995</span> I don't mind if it doesn't get accepted, that is an internal lint for <a href="https://github.com/trailofbits/dylint/">https://github.com/trailofbits/dylint/</a> ;)</p>



<a name="246874045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246874045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246874045">(Jul 22 2021 at 16:52)</a>:</h4>
<p>but getting back to the topic: yeah, I want to do some kind of global analysis, e.g. in a #[someattrmacro(someargs)] fn foo(some other args) be able to cross-match/validate the <code>someargs</code> with <code>some other args</code></p>



<a name="246875086"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246875086" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246875086">(Jul 22 2021 at 16:59)</a>:</h4>
<p>So I have to either use a <code>LateLintPass</code> to get to whatever the attribute macro produces, or, use an <code>EarlyLintPass</code> with a <code>register_pre_expansion_lint</code> and use its <code>check_attribute</code> to get this attribute macro; but then, I can't really access its arguments in a _relatively nice_ way, as its all in a <code>TokenStream</code></p>



<a name="246875310"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246875310" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246875310">(Jul 22 2021 at 17:00)</a>:</h4>
<p>There is the <code>syn</code> crate with <code>syn::parse(TokenStream)</code> function, but it doesn't operate on <code>rustc_ast::ast::tokenstream::TokenStream</code> but on <code>syn::TokenStream</code> (or sth like that)</p>



<a name="246875358"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246875358" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246875358">(Jul 22 2021 at 17:01)</a>:</h4>
<p>...so I am looking for sth similar to syn::parse but that works on TokenStream from ast :&lt;</p>



<a name="246878802"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246878802" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246878802">(Jul 22 2021 at 17:23)</a>:</h4>
<p>Oh, and when I tried using a <code>LateLintPass</code> for that, then, my problem is passing data between two lint passes. As in, registering two <code>LateLintPass</code> doesnt' seem to fire the lints in the order I register them, or, I would want the whole code to be linted by first lint and then the whole code to be linted by the second lint</p>



<a name="246878831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246878831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246878831">(Jul 22 2021 at 17:23)</a>:</h4>
<p>Instead of a given visitor/check function to be called by the first lint first and then by the second :/</p>



<a name="246881351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246881351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Cameron Steffen <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246881351">(Jul 22 2021 at 17:43)</a>:</h4>
<p>I wonder if you can accomplish what you are doing with one LateLintPass and use the check_*_post methods to do something after visiting.</p>



<a name="246887061"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246887061" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246887061">(Jul 22 2021 at 18:27)</a>:</h4>
<p>But isn't the order like:</p>
<p>check_fn<br>
check_expr (expr within fn)<br>
check_expr_post<br>
check_fn_post</p>



<a name="246890252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246890252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Cameron Steffen <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246890252">(Jul 22 2021 at 18:53)</a>:</h4>
<p>yes</p>



<a name="246890512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246890512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Cameron Steffen <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246890512">(Jul 22 2021 at 18:55)</a>:</h4>
<p>and there's check_crate_post</p>



<a name="246902690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246902690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246902690">(Jul 22 2021 at 20:45)</a>:</h4>
<p>Oh I see, nice</p>



<a name="246902703"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/257328-clippy/topic/ast%3A%3Atokenstream%3A%3ATokenStream%20parsing/near/246902703" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Disconnect3d <a href="https://rust-lang.github.io/zulip_archive/stream/257328-clippy/topic/ast.3A.3Atokenstream.3A.3ATokenStream.20parsing.html#246902703">(Jul 22 2021 at 20:45)</a>:</h4>
<p>Will try this later, thanks :)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>